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A  laiv*  and  Important  elaaa  of  variational  problama  have 
the  following  fom.  Olven  a  vector  equation  of  the  fom 

dx/dt  -  g(x,y),  x(0)  -  c, 

where  x  la  an  N-dlmenelonal  vector,  we  wlah  to  determine  an 
m-dlaenalonal  vector  y  so  aa  to  minimise  a  given  criterion 
functional 

J(y)  h(x,y)dt. 

0 

where  h(x,y)  Is  a  given  scalar  function. 

An  has  been  shown  In  some  recent  publications,  a  variety 
of  problems  of  this  nature  arising  In  economic  and  engineering 
control  processes  may  be  solved  computationally  by  combining 
the  theory  of  dynamic  programming  with  modem  digital  computers. 

In  recent  years,  problems  of  less  explicit  nature  have 
become  more  frequent.  Thus,  for  example,  what  Is  called  the 
"bang-bang”  control  problem  requires  that  y  be  chosen  so 
that  the  system  tends  to  a  specified  equilibrium  state  as 
rapidly  as  possible. 

The  upper  limit  of  Integration  Is  thus  not  predetermined, 
but  rather  a  function  of  the  choice  of  the  vector  y.  In  place 
of  a  formulation  In  precise  analytic  terms,  we  encounter  an 
Implicit  criterion  of  the  following  type: 

"When  X  satisfies  a  set  of  conditions  C^«C2«...,C^ 
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for  the  first  time,  we  want  a  given  scalar  function  of  x  to 
be  as  small  as  possible." 

A  particular  example  of  a  problem  of  this  nature ,  equiva¬ 
lent  to  one  we  shall  discuss  In  more  detail.  Is  one  In  which 
we  require  that  a  preassigned  function  be  a  minimum  for  the 
first  value  of  T  for  which  x^(T)  >  a^,  a  given  value. 


r-iyr- 
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ON  TOE  APPLICATION  :>?  CTOAMIC  PPOGRAMMINO 
TO  A  CLASS  OP  IMPLICIT  VARIATIONAL  PROBLEMS 

Richard  Bellman* 

John  M.  Richardson** 


1.  Introduction 

A  large  and  Important  class  of  variational  problems  have 
the  following  form.  Given  a  vector  equation  of  the  form 

(1)  ^  -  g(x,y),  x(0)  -  c, 

where  x  la  an  N— dimensional  vector,  we  wish  to  determine  an 
m— dlmendlonal  vector  y  so  as  tc  minimize  a  given  criterion 
functional 

(2)  J(y)  -  h(x,y)dt, 

•0 

where  h(x,y)  Is  a  given  scalar  function. 

TV»e  vector  y  may  be  subject  to  constraints  of  the  form 

(3)  r^(x,y)  <  0,  1  -  1 ,q. 

In  problems  Involving  "terminal  control,"  we  meet  the 
problem  of  minimizing  a  function  only  of  the  final  state 

(^)  I(y)  -  l<(x(T)). 

A  problem  of  this  nature  occurs  when  we  wish  to  have  the 

- 1 - 

The  RAND  Corporation. 

•  « 

Hughes  Aircraft  Company. 
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in  SOM  specif Isd  state  Xq(T)  at  timo  T,  without 
earing  how  the  systen  gets  there.  This  Is  usually  an  Ideall— 
Mtlon,  In  the  senae  that  a  more  realistic  problem  will 
Involve  a  coaiblnatlon  of  a  criterion  of  the  type  appearing  In 
(2)  together  with  some  Masure  of  the  value  of  the  final 
state. 

As  has  been  shown  In  som  recent  publications,  cf.  [l] 
where  further  references  may  be  found,  a  variety  of  problems 
of  this  nature  arising  In  economic  and  engineering  control 
procesaes  may  be  solved  computationally  by  combining  the 
theory  of  dynamic  programsilng  with  modem  digital  computers. 

In  recent  years,  problems  of  less  explicit  nature  have 
becoM  more  frequent.  TYius,  for  example,  what  Is  called  the 
"bang— bang”  control  problem  requires  that  y  be  chosen  so 
that  the  system  tend  to  a  specified  equilibrium  state  as 
rapidly  as  possible;  cf.  f2]  . 

The  upper  limit  of  Integration  Is  thus  not  predetemlned, 
but  rather  a  function  of  the  choice  of  the  vector  y.  In 
place  of  a  forvnilatlon  In  precise  analytic  terms  of  the  type 
appearing  In  (2)  or  (4),  we  encounter  an  Implicit  criterion  of 
the  following  type : 

"When  X  satisfies  a  set  of  conditions  C. ,C.,...,C 

1  z  p 

for  the  first  time,  we  want  a  given  scalar  ftinctlon  of  x  to 
be  as  small  as  possible." 

A  particular  example  of  a  problem  of  this  nature,  equiva¬ 
lent  to  one  we  shall  discuss  In  more  detail  below,  la  one  In 
which  we  roqxiire  that  a  preassigned  function  be  a  minimum  for 
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the  first  value  of  T  for  *fhlch  x,(T)  -  a,,  a  given  value. 

X  I 

A  number  of  quite  Interesting  existence  and  uniqueness 
questions  arise  In  conjunction  with  problem  statements  of  the 
foregoing  kind.  T^ese  will  be  discussed  at  some  time  In  the 
future.  Here  we  are  Interested  In  describing  a  technique 
which  can  be  used  to  obtain  computational  solutions  via  the 
functional  equation  path  of  dynamic  programming. 

'IVie  problem  becomes  of  even  more  Interesting  nature  If  we 
Insert  some  stochastic  Influences  Into  the  process.  Let  the 
governing  equation  be 

(5)  ^  -  g(x,y,r),  x(0)  -  c, 

where  r  Is  a  random  vector.  We  now  wish  to  minimize  an 
expected  deviation,  or  say  the  probability  that  the  deviation 
exceeds  a  given  critical  value. 

Once  again,  let  us  point  out  that  the  rigorous  groundwork 
for  th.ese  questions  remains  to  be  laid.  However,  as  we  shall 
see  below,  vc  have  a  simple  method  for  postponing  this  type  of 
Investigation . 

As  Is  to  be  expected,  certain  simplifications  are 
possible  if  tiiC  underlying  equations  are  linear,  l.e.  of  the 
f  orm 

(6)  x,,-Ax  -fy  ■♦■r,  •  c , 

'  ’  n-fl  n  •'n  n  0  ' 

and  the  criteria  quadratic.  We  shall  discuss  these  cases  In 
some  detail  since  they  are  of  some  Importance  Iri  connection 
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wlth  the  application  of  the  method  of  cucce3f?lve  approxlma*"  '•ns. 

Throughout,  our  aim  will  be  to  Illustrate  the  applicability 
of  the  functional  equation  technique  of  dynamic  programming  to 
the  computational  solution  of  questions  of  this  kind  which 
appear  In  many  ways  to  be  outside  the  domain  of  the  classical 
calculus  of  variations. 

2.  Preliminaries 

Since,  as  mentioned  above,  we  are  primarily  Interested  In 
a  computational  solution  of  Implicit  variational  problems  of 
the  type  described  In  the  foregoing  section,  we  shall  pose  our 
problem  In  discrete  terms.  Tlie  recurrence  relations  we  derive 
will  then  be  ready  for  use  In  a  digital  computer. 

In  place  of  the  differential  relation  of  (1.4),  consider 
the  difference  equation 

^n+1  "  ^^^^n’^n'^n^ '  ^0  “  ^ '  "  "  0,1,.. .,N. 

One  of  the  advantages  of  formulating  problems  In  this  fashion 
Is  that  there  are  now  no  conceptual  difficulties  concerning 
the  meaning  of  random  functions  or  the  existence  of  minimizing 
functions.  In  return,  sometime  or  other  we  must  show  that  the 
limit  of  the  discrete  process  exists,  and,  preferably,  yields 
the  continuous  process.  For  a  start  In  this  direction,  see 
. 

In  order  to  Illustrate  the  method  In  simple  fashion,  we 
shall  consider  a  two-dimensional  process. 
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(2)  x^(n-*'l)  -  x^(n)  -  y^(n)  -  r^(n),  x^(0)  -  c^. 


X2(n4l)  -  g^(xj^(n)  ,x^(n)  ,y^(n)  ,r^(n)) ,  x^(0)  -  c^. 

'Hie  aim  of  the  proceaa  la  to  choose  y^(n)  and  y^(n),  aub— 
Ject  to  conatrainta  of  the  fom 

(3)  ®  <  ^2'  ^  ^  ^2^^^  -  ^2 

2 

so  as  to  minimize  the  expected  value  of  (x^Cm)  —  x^)  where 
m  la  the  "time"  at  which  x^(m)  -  0.  The  r^(n)  are  Inde¬ 
pendent  random  variables  with  given  distributions. 

TVie  expected  value  la  over  the  random  variables  r^  and 
r^,  where  r^  can  depend  upon  the  choice  of  y^  and  y^, 
but.  In  any  case  la  subject  to  the  condition  that 

(4) 


It  follows  that  Xj^(n)  la  steadily  decreasing  as  n  Increases. 

lYie  recurrence  relation  In  (l)  Is  valid  until  x^(n)  -  0. 
Properly,  we  should  write 


(5) 


x^(n>l ) 


Maix 


0,  x^(n)  -  y^(n)  -  r^(n) 


TYie  process  ends  as  soon  as  assumes  the  value  zero. 

3.  Punctlonal  Equations 

It  Is  clear  that  the  minimum  of  the  expected  value  of 

2 

(x^Cn)  —  Xq)  depends  upon  c^  aind  and  only  upon  these 

variables  assuming  all  other  functions  and  dlstrll  utlons  known 
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and  fixed.  Let  us  then  write 

(1)  f(c,,c^)  -  Min  Exp  (x^(n)  -  x^)^. 

yi  r 


W©  have 

(2)  fCO.Cj)  '  (cj  - 


amd  the  principle  ol  optimality,  see  [l]  ,  yields  the 
functional  equation 

(3)  f(c,,c.)  -  Min 

yi.y^ 


Exp  f(c^  ~  ^1  “  . 


There  Is  no  difficulty  In  treating'  the  case  In  which  the 
distribution  of  random  effects  depends  upon  the  decisions  that 
are  made. 


4.  Probability  of  Deviation 

In  place  of  mean— square  deviation,  let  us  consider  the 
problem  of  determining  y^  amd  yp  so  as  to  minimize  the 
probability  that  1 Xp  -  Xq|  >  d. 

As  above,  let 


(1) 


Min  Prob 
^1 


|Xp  -  x^l  >  d 


Then 

(2)  f(0,cp)  >  1,  'Cp  -  XqI  >  d, 

■  -  *0' 
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«hll«  •atlsfl«a  the  sum  functional  eqviatlon  aa  in 

(3.3). 

5.  Diaottsaiwi  of  Ccaputatlonal  Solution 

In  order  to  detamlne  the  fimctlon  fCc^^Cg)  ualng  a 
digital  eoaqputar,  we  eBg>loy  a  dlacrete  grid  In  (c^,C2)-apace. 
Let  aaauM  only  the  aequence  of  valuea  0,5,26,...,  and 
*  aequence  of  valuea  0,£^,2r^, ...  .  Since  c^  la  mono- 
tonlcally  deereaalng  aa  the  proceaa  contlnuea,  we  can  uae  It 
aa  a  ”tlaa"  variable.  Vrlte 

(1)  f(k6,c2)  •  IjjCcg). 

Then  (3.3)  way  be  written 

(2)  ^k^®2^  "  ****  fp(8(k6,C2,y2,r2))  , 

^1*^2  L**1'**2 

where  p  la  detemlned  by  the  'condition 

(3)  P  -  [(cj  -  yj  -  *'i)/®]» 

the  greateat  Integer  contained  In  (c^  •>  y^^  »  r^)/6. 

Since  g(k6,c2>y2*i'2^  ^  general  will  not  be  an  Integral 

■ultlple  of  we  can  either  take  aa  Ita  value  the  neareat 
Integer  aultlple  of  aa  we  did  In  (3)«  or  we  can  uae  Inteiv. 
polatlon.  If  more  accurate  reaulta  are  dealred. 

The  value  of  fQ(c2)  la  determined  by  the  relation 

^0^®2^  "  (®2  ”  *0^  * 


(4) 
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Con8«<|u«ntly  (2)  furnishes  a  recurrence  relation  which  enables 
us  to  compute  the  function  textns  of 

n  •  0,1, . . .  ,1&>1.  Ve  th\i8  have  a  feasible  computational 

scheme. 

6.  Detenslnlstlc  Process 

Returning  to  a  purely  detetvlnlstlc  process,  as  speci¬ 
fied  by  (1.1),  we  may  wish  to  determine  y  so  that  x  is  Ir. 
some  desired  state  at  some  subsequent  time.  One  way  of 
attacking  this  problem  Is  to  tr^at  the  problem  of  minimising 
(x^CT)  —  Xq)^  where  T  Is  the  first  time  at  which  x^(T) 
has  Its  desired  value.  The  functional  equations  are  as  above, 
without  the  averaging  over  the  random  behavior. 

7.  Linear  Iquatlone  and  Quadratic  Criteria 

In  general,  the  application  of  a  straightforward 
functional  equation  approach  Is  limited  by  dimensionality 
difficulties  In  the  sense  that  functions  of  three  or  more 
variables  cannot  be  readily  stored  In  a  fast  memory.  Conse¬ 
quently,  the  techniques  described  above  must  be  aided  and 
abetted  by  successive  approximations  of  various  types,  a 
subject  which  has  been  discussed  elsewhere.  If,  however,  the 
guiding  equations  are  linear,  and  the  criteria  function  qua<l- 
ratlc,  then  the  sequence  of  functions  will  consist 

of  a  sequence  of  quadratic  functions  In  c.  These  functions 
are  determined  once  the  coefficients  are  determined.  As  we 
shall  see,  reasonably  8lq;>le  recurrence  relations  exist  ooiv> 
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n«cting  the  oo«fflcl«nt0  of  f„(c)  with  thoto  of 

Consldor,  to  bogin  with,  the  problon  of  choosing  the 
so  ss  to  ■InlBlse  the  expected  assn-equere  devistion 


(1) 


^•(y) 


Bxp 

r 


(x(T)  -  s,x(T)  -  s) 

\r»  By^) 


Here  T  ssswes  the  wslties  0,1,2,...,  B  is  a  positive 
definite  satrix,  s  is  a  specified  state  vector,  x  and  y 
are  related  by  aeans  of  the  linear  relations 

(*)  Vfl  •  **n  *  >'n  ♦  •'n'  *0  ‘  <=> 


Where  .^r^|  is  a  set  of  independent,  randoo  vectors  with 
Identical  distributions. 

The  process  is  assusMd  to  proceed  in  the  following 
fashion.  Ve  observe  c,  the  initial  state  and  on  this  basis 
and  the  foregoing  infomation,  choose  y^,  the  initial  control 
vector.  Then  a  randoa  effect  r^  occurs,  yielding  by  way  of 
(2)  a  new  state  vector  Ac  ♦  y^  4  r^.  The  process  then  eor^ 
tinues  in  this  way,  stage-by— stage,  a  "feedback  control* 
process . 

Although  this  problen  can  be,  and  has  been,  treated  by 
straightforward  variational  techniques,  we  shall  treat  it  by 
functional  equation  Methods.  There  is  soae  nerit  in  doing 
this  even  in  this  case,  and  in  addition  we  shall  prepare  the 
way  for  the  following  section  devoted  to  a  process  of  randosi 
duration. 
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Define  the  new  sequence  of  fvtnctlons  if^(c)l  by  means 
of  the  relation 

(3)  f-(c)  -  Min  J-(y). 

fyj 

Then 

(4)  -  (c  -  a,c  -  a), 

and  the  principle  of  optimality  yields  the  recurrence  relation 

(5)  f„(c)  -  Min  Exp  [(yQ.9yo)  ♦  ^0  ■*“  **0^]  ' 

^0  *'o 

for  n  ■  1,?, . . .  . 

Let  us  now  show  Inductively  that  each 
written  In  the  form 

(6)  ”*■  Si’ 

The  result  Is  obviously  so  for  n  •  0. 

Substituting  In  (5)*  we  have 

(7)  fp(c)  -  Min  Exp  r(yo»Byo)  *  ^0  ^  ^0  **0^^ 

♦  2(b^j,Ac  ♦  ♦  r,,)  +  . 

Taking  expected  values  and  using  the  result  that 
(0)  Min  [^(y,Cy)  ♦  2(g,y)J  -  -  (g,cr^g). 
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wh«n«v«r  C  ta  poaitlva  daflnit*,  we  eee  that  P„(c)  has  the 
fom  stated  in  (6).  Carrying  through  the  calculations,  we 
obtain  reeurrenee  relations  connecting  and  d^  with 

*Si-l»  ^D-1  "***  ‘^n-l* 

8,  Linear  Process  of  Randon  Ihiratlon 

Consider  now  a  system  specified  by  the  equations 

"n+l  •  %  -  ‘•in'  “0  •  'O' 

“Wl  •  **n  ♦  ‘‘n  ♦  ••n'  *0  ‘  '' 

Where  u^  and  r^  are  scalars,  and  r^  vectors. 

The  process  ends  whenever  u^  becosMS  sero  or  negative. 

The  quantity  r^^  Is  a  uniformly  positive  random  vari¬ 
able,  so  that  the  process  Is  always  finite.  The  control 
vectors  y^  are  to  be  chosen  so  as  to  minimise  the  expected 
value  of 

(2)  J(y)  -  (x(m)  -  a,x(m)  -  a)  ♦  J 

k-0  “ 

where  m  Is  Itself  a  random  variable  detenslned  by  the 
condition  that  It  Is  the  first  Integer  for  which  U|^  Is 
negative  or  sero. 

Write 


(3)  f(cQ,c)  -  Hln  Ixp  J(y). 

J  r 


Then 


P-1374 

5-14-58 

-12- 


(4)  f(0,c)  -  (c  -  a,c  -  a), 
and 

(5)  f(cQ,c)  -  Min  Sxp  f(cQ  -  ”*■  ^0  *'o^* 

y©  **0 

AaaxjDe,  as  previously,  that  can  assvnne  only  a 

discrete  set  of  values  with  a  similar  condition  on  r^^^.  Let, 
suitably  normalized,  Cq  take  the  values  0,1,...,  and  r^^ 
only  the  range  of  values  d^,d^-fl, . . .  ,d2>  nien,  writing 

(6)  f(k.c)  a  fj^(c),  k-  0,1,2,..., 
we  may  write  (5)  In  the  form 

I  1 

(7)  f„(c)  .  Min  Sxp  ■  J  ♦  '■oV  ' 

Xq  I 

Where 

(8)  ^1  “  probability  that  r^Q  •  1. 

The  function  f^(c)  Is  Identically  zero  for  k  £  0. 

Once  again.  It  Is  easy  to  see  that  each  element  of  the 
sequence  ^f|^(c)  Is  a  quadratic  function  of  c,  of  the  for* 

(9)  ^k^°'  "  "*■  2(bj^,c)  ♦  Uj^. 

The  recurrence  relations  connecting  M^,  b^,  u^  with 
bj^l*  can  be  obtained  from  (7)  In  the  way  Indicated 


above . 
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